Bootstrap Methods for the Cost - Sensitive Evaluation of Classi ersDragos

Many machine learning applications require classiiers that minimize an asymmetric cost function rather than the misclassiication rate, and several recent papers have addressed this problem. However, these papers have either applied no statistical testing or have applied statistical methods that are not appropriate for the cost-sensitive setting. Without good statistical methods, it is dii-cult to tell whether these new cost-sensitive methods are better than existing methods that ignore costs, and it is also diicult to tell whether one cost-sensitive method is better than another. To rectify this problem, this paper presents two statistical methods for the cost-sensitive setting. The rst constructs a conndence interval for the expected cost of a single classiier. The second constructs a con-dence interval for the expected diierence in costs of two classiiers. In both cases, the basic idea is to separate the problem of estimating the probabilities of each cell in the confusion matrix (which is independent of the cost matrix) from the problem of computing the expected cost. We show experimentally that these bootstrap tests work better than applying standard z tests based on the normal distribution.